Utforsking av overgangen fra tradisjonell dataanalyse til metoder med maskin- og dyp læring

Data analysis methods based on machine- and deep learning approaches are continuously replacing traditional methods. Models based on deep learning (DL) are applicable to many problems and often have better prediction performance compared to traditional methods. One major difference between the traditional methods and machine learning (ML) approaches is the black box aspect often associated with ML and DL models. The use of ML and DL models offers many opportunities but also challenges. This thesis explores some of these opportunities and challenges of DL modelling with a focus on applications in spectroscopy. DL models are based on artificial neural networks (ANNs) and are known to automatically find complex relations in the data. In Paper I, this property is exploited by designing DL models to learn spectroscopic preprocessing based on classical preprocessing techniques. It is shown that the DL-based preprocessing has some merits with regard to prediction performance, but there is considerable extra effort required when training and tuning these DL models. The flexibility of ANN architecture designs is further studied in Paper II when a DL model for multiblock data analysis is proposed which can also quantify the importance of each data block. A drawback of the DL models is the lack of interpretability. To address this, a different modelling approach is taken in Paper III where the focus is to use DL models in such a way as to retain as much interpretability as possible. The paper presents the concept of non-linear error modelling, where the DL model is used to model the residuals of the linear model instead of the raw input data. The concept is essentially a shrinking of the black box aspect since the majority of the data modelling is done by a linear interpretable model. The final topic explored in this thesis is a more traditional modelling approach inspired by DL techniques. Data sometimes contain intrinsic subgroups which might be more accurately modelled separately than with a global model. Paper IV presents a modelling framework based on locally weighted models and fuzzy partitioning that automatically finds relevant clusters and combines the predictions of each local model. Compared to a DL model, the locally weighted modelling framework is more transparent. It is also shown how the framework can utilise DL techniques to be scaled to problems with huge amounts of data.Metoder basert på maskin- og dyp læring erstatter i stadig økende grad tradisjonell datamodellering. Modeller basert på dyp læring (DL) kan brukes på mange problemer og har ofte bedre prediksjonsevne sammenlignet med tradisjonelle metoder. En stor forskjell mellom tradisjonelle metoder og metoder basert på maskinlæring (ML) er den "svarte boksen" som ofte forbindes med ML- og DL-modeller. Bruken av ML- og DL-modeller åpner opp for mange muligheter, men også utfordringer. Denne avhandlingen utforsker noen av disse mulighetene og utfordringene med DL modeller, fokusert på anvendelser innen spektroskopi. DL-modeller er basert på kunstige nevrale nettverk (KNN) og er kjent for å kunne finne komplekse relasjoner i data. I Artikkel I blir denne egenskapen utnyttet ved å designe DL-modeller som kan lære spektroskopisk preprosessering basert på klassiske preprosesseringsteknikker. Det er vist at DL-basert preprosessering kan være gunstig med tanke på prediksjonsevne, men det kreves større innsats for å trene og justere disse DL-modellene. Fleksibiliteten til design av KNN-arkitekturer er studert videre i Artikkel II hvor en DL-modell for analyse av multiblokkdata er foreslått, som også kan kvantifisere viktigheten til hver datablokk. En ulempe med DL-modeller er manglende muligheter for tolkning. For å adressere dette, er en annen modelleringsframgangsmåte brukt i Artikkel III, hvor fokuset er på å bruke DL-modeller på en måte som bevarer mest mulig tolkbarhet. Artikkelen presenterer konseptet ikke-lineær feilmodellering, hvor en DL-modell blir bruk til å modellere residualer fra en lineær modell i stedet for rå inputdata. Konseptet kan ses på som en krymping av den svarte boksen, siden mesteparten av datamodelleringen er gjort av en lineær, tolkbar modell. Det siste temaet som er utforsket i denne avhandlingen er nærmere en tradisjonell modelleringsvariant, men som er inspirert av DL-teknikker. Data har av og til iboende undergrupper som kan bli mer nøyaktig modellert hver for seg enn med en global modell. Artikkel IV presenterer et modelleringsrammeverk basert på lokalt vektede modeller og "fuzzy" oppdeling, som automatisk finner relevante grupperinger ("clusters") og kombinerer prediksjonene fra hver lokale modell. Sammenlignet med en DL-modell, er det lokalt vektede modelleringsrammeverket mer transparent. Det er også vist hvordan rammeverket kan utnytte teknikker fra DL for å skalere opp til problemer med store mengder data

Existence and detectability of very fast oscillations in spiking network models

Extremely high frequency oscillations are present in dynamics of different simulated neuronal networks, but are not yet observed in experimental recordings. This raises the question about the origin of these high frequency oscillations. Focusing on the microcircuit model of an area in the visual cortex, it is shown that the oscillations, visible as vertical stripes in raster plots from population activities, are related to peaks in the power spectra of the population-averaged firing activities. It is shown that the oscillations are not caused by the time discretization of the spike detection in the simulation of the network or the use of discretized delay values. Given the difference in the number of neurons between experimental recordings and the populations in the network model, a subsampling of neurons from the network model with different sample sizes was performed to analyse the oscillations for subsamples of neurons with similar sizes as the experimental recordings. The results show that peaks in the power spectra can be observed for subsamples of few hundred neurons from a population, given that the whole population shows sufficient oscillatory activity. Analytical results for the power spectra of the microcircuit model described by Bos et al. [2016] show that the choice of delay distribution for the inhibitory connections in the model has a large effect on the peaks. This network model originally uses a truncated Gaussian distribution for the synaptic delays. The use of an exponential delay distribution shows a spectrum with no peaks for frequencies above 100 Hz for any of the populations. Analytical power spectra from models using uniform and lognormal distribution for the delays have peaks in the power spectra around 80 Hz and 300 Hz, similar to the spectra from the model using a truncated Gaussian distribution. The amplitudes of the peaks are not the same for the models with different delay distributions. This is explained by the different contributions of each eigenmode obtained from the connectivity matrix of the network. It is also shown that varying the parameters of the delay distribution greatly shifts or scales both the low and high frequency peaks, making it possible to tune the parameters to manipulate the oscillations of the network dynamics. The dynamics of area V1 from the multi-area model are analyzed for oscillations. The results show that the oscillatory dynamics are almost completely gone.Ekstremt høyfrekvente oscilleringer er tilstedeværende i dynamikken til ulike simulerte nevronale nettverk, men har enda ikke blitt observert i eksperimentelle målinger. Dette fører til spørsmålet om opphavet til disse høyfrekvente oscilleringene. Ved å fokusere på ”microcircuit”-modellen for et område i synssenteret i hjernen vises det at oscilleringene, synlig som vertikale striper i raster plott fra populasjonsaktiviteten, kan kobles til topper i ”power”-spekteret til den populasjons-gjennomsnittlige fyringsaktiviteten. Det vises at oscilleringene ikke oppstår på grunn av diskrete tidssteg i simuleringen av nettverket eller å bruke diskrete ”delay”-verdier. På grunn av forskjellen mellom antall nevroner i eksperimentelle målinger og størrelsen på populasjonene i nettverksmodellen, en ”subsampling” av nevroner med ulike utvalgsstørrelser ble gjennomført for å analysere oscilleringene for utvalg med et antall nevroner i samme størrelsesorden som i de eksperimentelle målingene. Resultatene viser at toppene i ”power”-spekteret kan bli observert for så lite som noen få hundre nevroner, gitt at populasjonen viser tilstrekkelig oscillatorisk aktivitet. Analytiske resultater av ”power”-spektrene til ”microcircuit”-modellen, som forklart av Bos et al. [2016], viser at valget av ”delay”-fordelingen til de inhibitoriske koblingene i modellen har en stor effekt på toppene. Denne nettverksmodellen bruker opprinnelig en trunkert Gaussfordeling for de synaptiske ”delayene”. Ved å bruke en eksponentiell fordeling til ”delayene” observeres det ingen topp i de analytiske ”power”-spektrene for frekvenser over 100 Hz. De analytiske ”power”-spektrene for modeller med uniform- og lognormalfordelinger til ”delayene” viser topper både for rundt 80 Hz og 300 Hz, tilsvarende resultatene fra å bruke modellen med den originale trunkerte Gaussfordelingen. Amplituden til på toppene i spektrene er forskjellig for modellene med de ulike ”delay”-fordelingene som kan forklares ved de ulike bidragene til spektrene fra de ulike eigenmodene tilhørende koblingsmatrisen til nettverket. Det er også vist at variasjon i parametrene til delayfordelingene har en stor påvirkning på både posisjonen og amplituden til toppene i ”power”-spektrene. Dette gjør det mulig å tune parameterne for å manipulere den oscillatoriske dynamikken i nettverket. Dynamikken til området V1 av synssenteret i hjernen fra ”multi-area”-modellen er analysert for oscilleringer.M-D

Existence and detectability of very fast oscillations in spiking network models

Extremely high frequency oscillations are present in dynamics of different simulated neuronal networks, but are not yet observed in experimental recordings. This raises the question about the origin of these high frequency oscillations. Focusing on the microcircuit model of an area in the visual cortex, it is shown that the oscillations, visible as vertical stripes in raster plots from population activities, are related to peaks in the power spectra of the population-averaged firing activities. It is shown that the oscillations are not caused by the time discretization of the spike detection in the simulation of the network or the use of discretized delay values. Given the difference in the number of neurons between experimental recordings and the populations in the network model, a subsampling of neurons from the network model with different sample sizes was performed to analyse the oscillations for subsamples of neurons with similar sizes as the experimental recordings. The results show that peaks in the power spectra can be observed for subsamples of few hundred neurons from a population, given that the whole population shows sufficient oscillatory activity. Analytical results for the power spectra of the microcircuit model described by Bos et al. [2016] show that the choice of delay distribution for the inhibitory connections in the model has a large effect on the peaks. This network model originally uses a truncated Gaussian distribution for the synaptic delays. The use of an exponential delay distribution shows a spectrum with no peaks for frequencies above 100 Hz for any of the populations. Analytical power spectra from models using uniform and lognormal distribution for the delays have peaks in the power spectra around 80 Hz and 300 Hz, similar to the spectra from the model using a truncated Gaussian distribution. The amplitudes of the peaks are not the same for the models with different delay distributions. This is explained by the different contributions of each eigenmode obtained from the connectivity matrix of the network. It is also shown that varying the parameters of the delay distribution greatly shifts or scales both the low and high frequency peaks, making it possible to tune the parameters to manipulate the oscillations of the network dynamics. The dynamics of area V1 from the multi-area model are analyzed for oscillations. The results show that the oscillatory dynamics are almost completely gone

Ultra-high frequency spectrum of neuronal activity

The activity of spiking network models exhibits fast oscillations (>200 Hz), caused by inhibition-dominated excitatory-inhibitory loops [1, 2]. As correlations between pairs of neurons are weak in nature and models, fast oscillations have so far received little attention.Today’s models of cortical networks with natural numbers of neurons and synapses [3] remove any uncertainty about down-scaling artifacts [4]. Fast oscillations here arise as vertical stripes in raster diagrams. We discuss experimental detectability of oscillations, ask whether they are an artifact of simplified models, and identify adaptations to control them.The population rate spectrum decomposes into single-neuron power spectra (∼N) and cross-spectra of pairs of neurons (∼N2) [5,6]. For low numbers of neurons (100) and weak correlations, the single-neuron spectra dominate the compound spectrum. Coherent oscillations in the population activity may thus go unnoticed in experimental spike recordings. Population measures obtained from large neuron ensembles (e.g., LFP), however, should show a pronounced peak.Cortical network models allow an investigation from different angles. We rule out artifacts of time-discrete simulation and investigate the effect of distributed synaptic delays: exponential distributions decrease the oscillation amplitude, expected by their equivalence to low-pass filtering [7], whereas truncated Gaussian distributions are ineffective.Surprisingly, a model of V1 [8], with the same architecture, but fewer synapses per neuron, does not exhibit fast oscillations. Mean-field theory shows that loops within each inhibitory population cause fast oscillations. Peak frequency and amplitude are determined by eigenvalues of the effective connectivity matrix approaching instability [9]. Reducing the connection density decreases the eigenvalues, increasing their distance to instability; we thus expect weaker oscillations.Counter to expectation and simulation, mean-field theory predicts an increase, explained by an overestimation of the transfer function at high frequencies [10]: the initial network appears to be linearly unstable, with |λ|>1; reduced connectivity seemingly destabilizes the system. A semi-analytical correction restores qualitative agreement with simulation.The work points at the importance of models with realistic cell densities and connectivity, and illustrates the productive interplay of simulation-driven and analytical approaches.References 1. Brunel, N. Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. JComputNeurosci 8, 183–208 (2000)., 10.1371/journal.pcbi.1006359 2. Brunel, N. & Wang, X.-J. What Determines the Frequency of Fast Network Oscillations With Irregular Neural Discharges? I. Synaptic Dynamics and Excitation-Inhibition Balance. JNeurophysiol 90, 415–430 (2003)., 10.1152/jn.01095.2002 3. Potjans, T. C. & Diesmann, M. The Cell-Type Specific Cortical Microcircuit: Relating Structure and Activity in a Full-Scale Spiking Network Model. CerebCortex 24, 785–806 (2014)., 10.1093/cercor/bhs358 4. van Albada, S. J., Helias, M. & Diesmann, M. Scalability of asynchronous networks is limited by one-to-one mapping between effective connectivity and correlations. ploscb 11, e1004490 (2015)., 10.1371/journal.pcbi.1004490 5. Harris, K. D., & Thiele, A. Cortical state and attention. Nature Reviews Neuroscience, 12(9), 509-523 (2011)., 10.1038/nrn3084 6. Tetzlaff, T., Helias, M., Einevoll, G. T., & Diesmann, M. Decorrelation of neural-network activity by inhibitory feedback. PLoS Comput Biol, 8(8), e1002596 (2012)., 10.1371/journal.pcbi.1002596 7. Mattia, M., Biggio, M., Galluzzi, A. & Storace, M. Dimensional reduction in networks of non-Markovian spiking neurons: Equivalence of synaptic filtering and heterogeneous propagation delays. PLoS Comput Biol 15, e1007404 (2019)., 10.1371/journal.pcbi.1007404 8. Schmidt, M. et al. A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas. ploscb 14, e1006359 (2018)., 10.1023/a:1008925309027 9. Bos, H., Diesmann, M. & Helias, M. Identifying Anatomical Origins of Coexisting Oscillations in the Cortical Microcircuit. ploscb 12, e1005132 (2016)., 10.1371/journal.pcbi.1005132 10. Schuecker, J., Diesmann, M. & Helias, M. Modulated escape from a metastable state driven by colored noise. Phys. Rev. E (2015)., 10.1103/PhysRevE.92.05211